[NV] feat(minimaxm3): add B200 disaggregated configs / [NV] feat(minimaxm3)：新增 B200 分离式配置 - #2230
[NV] feat(minimaxm3): add B200 disaggregated configs / [NV] feat(minimaxm3)：新增 B200 分离式配置#2230xinli-sw wants to merge 17 commits into
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| slurm: | ||
| account: coreai_comparch_inferencex | ||
| partition: batch | ||
| time_limit: "01:00:00" |
There was a problem hiding this comment.
🟡 All four new B200 minimax-m3 recipes set slurm.time_limit: "01:00:00", well below the launcher's 4h default and the 3-8h used elsewhere in the tree, and none define a health_check block, so the launcher's sed that bumps max_attempts 360→720 (meant to absorb cold-cache loads off shared FS) is a no-op here. The model loads from shared scratch FS, so a slow cold load could eat the 1h walltime and kill the job in TIMEOUT with no results — worth double-checking against the sibling B300 recipes (no time_limit override, health_check present) before merge.
Extended reasoning...
The bug: all four new recipes (2p2d-c128, 3p2d-c256, 3p2d-c512, 4p2d-c1024) set slurm.time_limit: "01:00:00". This overrides the generated srtslurm.yaml's default_time_limit: "4:00:00" (set in runners/launch_b200-dgxc.sh) downward to a quarter of that. Across the whole srt-slurm-recipes tree, the only other explicit time_limit values used are 3h and 8h — 1h is an outlier by a wide margin, and it's the only class of minimax-m3 recipe that sets it this low.\n\nWhy the launcher's usual safety net doesn't apply: runners/launch_b200-dgxc.sh has a sed step specifically to absorb slow cold-cache model loads off shared FS:\n\nThis only works if the recipe already has a health_check.max_attempts line to rewrite (going from 360×10s=3600s to 720×10s=7200s). None of the four new recipes define a health_check section at all, so this sed matches nothing and is a silent no-op. Compare to the sibling B300-fp4 minimax-m3 recipes (same checkpoint, same container image), which set no slurm.time_limit override (inheriting the 4h default) and still carry health_check.max_attempts: 360, letting the sed extend their health-check budget to 2h. The new B200 recipes get neither protection.\n\nWhy this matters here specifically: MODEL_PATH for minimaxm3+fp4 resolves to /scratch/fsw/models/MiniMax-M3-NVFP4, shared scratch FS — the same class of cold-cache-slow-load risk that motivated both the sed and the 8h time_limit/1440 max_attempts used by the sibling deepseek-v4 disagg recipes (comment there: "to absorb cold-cache model loads"). If a cold cache causes a slow multi-worker MoE weight load (the 4p2d/c1024 config spans 4 nodes and has the most startup coordination), the health-check window and remaining walltime could both be exhausted before srtctl ever runs the benchmark sweep, and SLURM kills the job in TIMEOUT — a multi-node B200 allocation consumed with zero results collected, which is a much worse failure mode than a clean error.\n\nProof sketch: launch job → sed looks for ^ max_attempts: [0-9]* in the recipe, finds no match (none of the 4 files have a health_check key), no-op → srtctl applies the recipe with its default health-check window (360×10s=3600s=1h) → model load from /scratch/fsw/models/MiniMax-M3-NVFP4 under cold cache takes long enough to approach that 1h health-check budget → SLURM's own 1h wall-clock limit (time_limit: "01:00:00") expires around the same time the load might still be settling → job terminates in TIMEOUT before the benchmark concurrency point runs → no results_concurrency_*.json produced, wasted allocation.\n\nCounterpoint worth weighing: a reviewer flagged that each of these recipes runs a single concurrency point (skip_initial_test: true, one value in concurrencies) rather than a multi-point sweep like the B300 sibling that relies on the 4h default, and that MiniMax-M3-NVFP4 is a much smaller 4-bit checkpoint than the ~680GB DSR1-FP8 case the sed comment references — so 1h may be a deliberate, proportionately-sized choice rather than an oversight, and the perf-changelog even labels these "validated" configurations. That's plausible, but it doesn't fully explain why these are the only minimax-m3 recipes with no health_check block at all (even the B300 siblings running fewer/no sweep points keep one), and a single occurrence of cold cache during CI reruns would still zero out the run. Given the WIP status and that this is easy to align with the siblings, it's worth a second look before merge.\n\nFix: either drop the slurm.time_limit override (inherit the 4h default like the B300 siblings) or add a health_check block (e.g. max_attempts: 360) so the launcher's sed can extend it, matching the pattern used everywhere else in the tree.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29458030349 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29461201009 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29463463002 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29468181344 |
64eef80 to
2ada754
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29471584525 |
79aa46c to
f345aa7
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29500473914 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29506626382 |
2 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29506626382 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29506626382 |
f0a7050 to
47f6115
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29650707821 |
47f6115 to
dab5d91
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29656178792 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29650707821 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29650707821 |
288eccd to
845cee3
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29673614296 |
845cee3 to
9c33842
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29951749216 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29980859669 |
…B200/B300 disagg configs - Merge origin/main (resolve perf-changelog.yaml conflict by keeping both entries) - Bump container image to vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9 across all 12 branch-specific minimax-m3 YAML configs - Replace UCX_TLS with UCX_TCP_AF_PRIO=inet; remove UCX_IB_ROCE_REACHABILITY_MODE in both prefill_environment and decode_environment of each config
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30053070865 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30098465522 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30098998886 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30101250351 |
xinli-sw
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30101250351 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30117917116 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30118047286 |
|
/reuse-sweep-run |
|
/reuse-sweep-run 30118047286 |
…skip-sweep] The config and all six b200-fp4 recipes pin vllm/vllm-openai:nightly-4080263bb2c5d10deac17aaeb88e0823bc35bca9, but the changelog entry still cited the pre-bump cu129-nightly-8e981630c9 tag. Documentation-only fix; no benchmark or config behavior changes.
|
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Additional detail section:
Signed: |
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — |
Ankur-singh
changed the title
3 participants
Summary
Validation
中文说明
验证